Tag: GPU kernels
3 reviews
Kernel optimization for LLM training: techniques, trade-offs, and use cases
Which kernel-level optimization techniques most improve LLM training efficiency, and what are the trade-offs and use cases of each?
Kernel-level optimization is where most LLM training efficiency gains of the last six years come from, and the field splits into five families: fused attention kernels, low-precision GEMM kernels, mixture-of-experts kernels, communication-overlap kernels, and fused elementwise/optimizer kernels. The evidence is consistent that each family delivers large, workload-specific gains — 2x on attention, 25-75% on FP8 training, 1.4-5.7x on MoE layers, 1.2-2.9x on long-context training — but the gains are conditional on hardware generation, model architecture, and engineering effort, and almost every headline number is vendor- or single-group-benchmarked. The main caveat: no shared benchmark exists, so cross-paper 'x-times faster' claims are not directly comparable.
GPU kernel development for LLM inference: Triton, TileLang, FlashMLA, and FlashInfer compared
How do Triton, TileLang, FlashMLA, and FlashInfer compare as GPU kernel development tools for LLM inference, and which use cases does each serve best?
The four tools fall into two families: kernel DSLs and compilers (Triton, TileLang) that generate kernels from Python-like tile programs, and attention kernel libraries and engines (FlashInfer, FlashMLA) that ship pre-built or JIT-compiled CUDA kernels for serving. The evidence shows DSLs are within reach of hand-tuned CUDA on GEMM and attention once tuned (59-105% of vendor baselines depending on workload and effort), while the libraries deliver 29-69% lower inter-token latency than compiler backends in serving benchmarks and dominate production deployment. FlashMLA is uniquely tied to one workload, DeepSeek's MLA, and has no peer-reviewed paper, so it is documented here through the DeepSeek technical reports and third-party kernel evaluations. On the training side the same DSLs power the ecosystem layer (torch.compile, Liger, MOSS's Triton FP8 GEMMs), but the training-kernel frontier — block-sparse MoE, single-kernel distributed MoE, FP8 training — is written in expert CUDA and CUTLASS, and FlashInfer and FlashMLA remain forward-only serving libraries. Confidence is moderate: almost all performance claims come from vendor-authored or single-group benchmarks with no independent head-to-head of all four tools.
Kernel fusion for efficient LLM training and inference
Which kernel-fusion methods improve the efficiency of large language model training and inference, and what gains do they deliver?
Kernel fusion — computing several operations in one GPU kernel to cut memory traffic and launch overhead — is the dominant mechanism behind the efficiency gains of the past six years. The evidence is consistent that attention kernels (FlashAttention and its descendants) deliver 2–4x speedups and near-constant memory in sequence length, that fused training-op kernels (Liger and peers) recover roughly 20% throughput and 60% memory on the training side, and that FP8 pipelines now train 10–75% faster with matched quality. Most evidence is preprint-grade and hardware-specific; the field lacks a shared benchmark, and the newest frontier is using LLMs themselves to write the kernels.